{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hackathon #2\n",
    "\n",
    "Written by Eleanor Quint\n",
    "\n",
    "Topics: \n",
    "- Dense layers\n",
    "- Gradient descent optimization\n",
    "- Training by minibatch/gradient step and epoch\n",
    "- TensorBoard\n",
    "\n",
    "This is all setup in a IPython notebook so you can run any code you want to experiment with. Feel free to edit any cell, or add some to run your own code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll start with our library imports...\n",
    "from __future__ import print_function\n",
    "\n",
    "import tensorflow as tf         # to specify and run computation graphs\n",
    "import numpy as np              # for numerical operations taking place outside of the TF graph\n",
    "import matplotlib.pyplot as plt # to draw plots\n",
    "\n",
    "mnist_dir = '/work/cse496dl/shared/hackathon/02/mnist/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A First Attempt at Classifying MNIST\n",
    "\n",
    "MNIST is a dataset of greyscale `28x28` handwritten digits labelled 1 through 9. We'll use it for a 10-class problem to learn the basics of classification.\n",
    "\n",
    "Let's have a look at the data first. We load the data from `numpy` save files using `np.load` and can visualize it with matplotlib's `plt.imshow`. The images (shape `[28,28]`) are flat when we first load them (shape `[784]`), so we'll have to [np.reshape](https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.reshape.html) any images we want to visualize. The labels are \"one-hot\", or arrays of length equal to the number of classes (in this case, 10) with the `n`-th entry set to 1 and the rest to 0, indicating the integer value `n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train image tensor shape: (55000, 784)\n",
      "Train image tensor shape: (55000, 10)\n",
      "Train image tensor shape: (10000, 784)\n",
      "Train image tensor shape: (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "# load our dataset, MNIST\n",
    "train_images = np.load(mnist_dir + 'mnist_train_images.npy')\n",
    "print(\"Train image tensor shape: \" + str(train_images.shape))\n",
    "\n",
    "train_labels = np.load(mnist_dir + 'mnist_train_labels.npy')\n",
    "print(\"Train label tensor shape: \" + str(train_labels.shape))\n",
    "\n",
    "test_images = np.load(mnist_dir + 'mnist_test_images.npy')\n",
    "print(\"Test image tensor shape: \" + str(test_images.shape))\n",
    "\n",
    "test_labels = np.load(mnist_dir + 'mnist_test_labels.npy')\n",
    "print(\"Test label tensor shape: \" + str(test_labels.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A label looks like this: [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "And an image looks like this:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADklJREFUeJzt3X+MHPV5x/HPY/t8NsYkOAbHNRC7EUmgVLHp1aC4QqaWET+imESB4LbERVEvSLgNKv1BrUimqtpahISQNhAdwcSJgiFVQnAUqwWdGrnIlPhMHWwwGEQNXGz5IIeC3cTm7Hv6x81Fh7n57rI7s7N3z/slnXZ3npmbR2t/bnb3Oztfc3cBiGdK1Q0AqAbhB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8Q1LRW7my6dfoMzWrlLoFQjur/9JYfs3rWbSr8Zna5pLskTZX0TXffkFp/hmbpIlvRzC4BJDzpvXWv2/DLfjObKunrkq6QdL6k1WZ2fqO/D0BrNfOef6mkF939JXd/S9KDklYV0xaAsjUT/gWSXh3zuD9b9jZm1m1mfWbWN6RjTewOQJGaCf94Hyq84/vB7t7j7l3u3tWhziZ2B6BIzYS/X9LZYx6fJelAc+0AaJVmwr9D0rlmtsjMpku6TtKWYtoCULaGh/rc/biZrZX0HxoZ6tvo7s8U1hmAUjU1zu/uWyVtLagXAC3E6b1AUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBNXSKbox+UxZnJ6b9eWr3ptbe/amu5PbPvqrjmT9js/+UbJu23+WrEfHkR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmpqnN/M9ks6LOmEpOPu3lVEU2gf094/L1m/5Ds7kvW/nPNcbm3I0/u+dOaJZP3Gz8xM1j/00/z/3n78eHrnARRxks+l7v56Ab8HQAvxsh8Iqtnwu6RHzWynmXUX0RCA1mj2Zf8ydz9gZmdKeszMnnP3bWNXyP4odEvSDJ3S5O4AFKWpI7+7H8huByQ9LGnpOOv0uHuXu3d1qLOZ3QEoUMPhN7NZZjZ79L6kyyTtKaoxAOVq5mX/PEkPm9no73nA3f+9kK4AlK7h8Lv7S5I+WmAvqED/330sWb/2up8k66lx/LI9/+mvJ+sX/vzPc2u/dfv2otuZcBjqA4Ii/EBQhB8IivADQRF+ICjCDwTFpbsngSmzZ+fWnt+QvrT2nlVfTdY7bGpDPbWDj336f3Jr+29vYSNtiiM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFOP8k8NzXPpRb23dZehpsaeKO49dy2Xufya3dveKa5LbTencW3U7b4cgPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzt8Gpp52WrK+944PJ+u7Vn4tUZ3eQEeTwydmvZFbe1/P/cltb1+6PFk/8YvBRlpqKxz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiComuP8ZrZR0sclDbj7BdmyOZIekrRQ0n5J17p7/qAqkmqN4++76hs1fkPcsfxGLZsxlF5hyuS9zsGoeo7835J0+UnLbpXU6+7nSurNHgOYQGqG3923STr5dKZVkjZl9zdJurrgvgCUrNH3/PPc/aAkZbdnFtcSgFYo/dx+M+uW1C1JM3RK2bsDUKdGj/yHzGy+JGW3A3krunuPu3e5e1eHOhvcHYCiNRr+LZLWZPfXSHqkmHYAtErN8JvZZklPSPqwmfWb2eckbZC00sxekLQyewxgAqn5nt/dV+eUVhTcy4Q1ZfbsZD11XX2p1vfxpSrH8b/0i/OT9dlTjybrnznt2dza2pc/kdx230Pp8x923PovyXozfvX7C5P1zq2vlbbvVuEMPyAowg8ERfiBoAg/EBThB4Ii/EBQXLq7AM9vSA+H1Z4mu7qhvC8O/F6yvvuP08OUwzM7kvX7l16ZWzvjnieS2867+EiyXqbOWw6mV9jamj7KxJEfCIrwA0ERfiAowg8ERfiBoAg/EBThB4JinL8AVy7dVXULuWqN4z99SfrryMOH9zW1/zN2NrU5SsSRHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCYpx/lFmy/MZnL86t3TD3rhq/vNzpntcPLMmt1fw+fpPj+JPVkW8sSNZPVX+LOikPR34gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCKrmOL+ZbZT0cUkD7n5Btuw2SX8maXSe4nXuPqGvZD54Q/44viRt/4d/TVTLHcev5WefWpRbO/G/jOM34j0/eSlZP9GiPspUz5H/W5IuH2f5ne6+OPuZ0MEHIqoZfnffJmmwBb0AaKFm3vOvNbOnzWyjmZ1eWEcAWqLR8N8j6YOSFks6KOnLeSuaWbeZ9ZlZ35CONbg7AEVrKPzufsjdT7j7sKR7JS1NrNvj7l3u3tWhzkb7BFCwhsJvZvPHPPykpD3FtAOgVeoZ6tssabmkuWbWL2m9pOVmtliSS9ov6fMl9gigBDXD7+6rx1l8Xwm9VOp9f/JK1S3kenGoxmclxyfmqPPU09OfE7/2xaOl7Xvf0FvpFXy4tH23C87wA4Ii/EBQhB8IivADQRF+ICjCDwTFpbvbwAWP35Csz3tgRrI+89WfFtlOoVLDecf+LT09+PbzNje1795fn5Jb2/AXNya37Xx9R1P7ngg48gNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzt4FFd3iy7jsqHMefUuOy5Et/J1k+Mi//HIXe8+5ppKO6PTx4YW6tc+vkH8evhSM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFOH8beOEL6X8GH7woWf/I+vxpuAc+9ZHktoMfTV+i2qelz0HYt6rcsfpmbPvxktzaOdrewk7aE0d+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiq5ji/mZ0t6duS3i9pWFKPu99lZnMkPSRpoaT9kq519zfKa3Xyeu7Sbza1/Y+veE9u7Xen/yi57TnTZja17zIdOvHrZH3lfX+TrH/gn/pya+mzF2Ko58h/XNIt7n6epIsl3WRm50u6VVKvu58rqTd7DGCCqBl+dz/o7k9l9w9L2itpgaRVkjZlq22SdHVZTQIo3rt6z29mCyUtkfSkpHnuflAa+QMh6cyimwNQnrrDb2anSvq+pJvd/c13sV23mfWZWd+QjjXSI4AS1BV+M+vQSPC/6+4/yBYfMrP5WX2+pIHxtnX3HnfvcveuDnUW0TOAAtQMv5mZpPsk7XX3r4wpbZG0Jru/RtIjxbcHoCz1fKV3maTrJe02s13ZsnWSNkj6npl9TtIrkq4pp8XWeO3Bc9IrrG9NH4246pRfJqrtO5RXy4rv/HWyvvDv01/LZTgvrWb43f1xSZZTXlFsOwBahTP8gKAIPxAU4QeCIvxAUIQfCIrwA0Fx6e7M3Hv/O1m/WGtza/evuzO57XkdHQ31NBm8MXw0t3bp3elx/EVfSk9Nzjh+czjyA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPOP8vSo8dyeJ3JrN/7y5uS2Q9cPJuvbl2xO1qt0ZDh96bU//Oe/StanHc1/Xs/ayPfxq8SRHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCMq8xvl2k02yOX2Rc7Rsoy5Peqzd9MO9S+2/DkR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgqoZfjM728z+08z2mtkzZvaFbPltZvZzM9uV/VxZfrsAilLPxTyOS7rF3Z8ys9mSdprZY1ntTne/o7z2AJSlZvjd/aCkg9n9w2a2V9KCshsDUK539Z7fzBZKWiLpyWzRWjN72sw2mtnpOdt0m1mfmfUNKX1JKACtU3f4zexUSd+XdLO7vynpHkkflLRYI68Mvjzedu7e4+5d7t7Voc4CWgZQhLrCb2YdGgn+d939B5Lk7ofc/YS7D0u6V9LS8toEULR6Pu03SfdJ2uvuXxmzfP6Y1T4paU/x7QEoSz2f9i+TdL2k3Wa2K1u2TtJqM1uskSss75f0+VI6BFCKej7tf1zSeN8P3lp8OwBahTP8gKAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQbV0im4ze03Sy2MWzZX0essaeHfatbd27Uuit0YV2dsH3P2MelZsafjfsXOzPnfvqqyBhHbtrV37kuitUVX1xst+ICjCDwRVdfh7Kt5/Srv21q59SfTWqEp6q/Q9P4DqVH3kB1CRSsJvZpeb2fNm9qKZ3VpFD3nMbL+Z7c5mHu6ruJeNZjZgZnvGLJtjZo+Z2QvZ7bjTpFXUW1vM3JyYWbrS567dZrxu+ct+M5sqaZ+klZL6Je2QtNrdn21pIznMbL+kLnevfEzYzC6RdETSt939gmzZ7ZIG3X1D9ofzdHf/2zbp7TZJR6qeuTmbUGb+2JmlJV0t6U9V4XOX6OtaVfC8VXHkXyrpRXd/yd3fkvSgpFUV9NH23H2bpMGTFq+StCm7v0kj/3laLqe3tuDuB939qez+YUmjM0tX+twl+qpEFeFfIOnVMY/71V5TfrukR81sp5l1V93MOOZl06aPTp9+ZsX9nKzmzM2tdNLM0m3z3DUy43XRqgj/eLP/tNOQwzJ3v1DSFZJuyl7eoj51zdzcKuPMLN0WGp3xumhVhL9f0tljHp8l6UAFfYzL3Q9ktwOSHlb7zT58aHSS1Ox2oOJ+fqOdZm4eb2ZptcFz104zXlcR/h2SzjWzRWY2XdJ1krZU0Mc7mNms7IMYmdksSZep/WYf3iJpTXZ/jaRHKuzlbdpl5ua8maVV8XPXbjNeV3KSTzaU8VVJUyVtdPd/bHkT4zCz39bI0V4amcT0gSp7M7PNkpZr5FtfhyStl/RDSd+TdI6kVyRd4+4t/+Atp7flGnnp+puZm0ffY7e4tz+Q9F+Sdksazhav08j768qeu0Rfq1XB88YZfkBQnOEHBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCo/wdJQeW1e8zgQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize some of the data\n",
    "idx = np.random.randint(train_images.shape[0])\n",
    "print(\"A label looks like this: \" + str(train_labels[idx]))\n",
    "print(\"And an image looks like this:\")\n",
    "imgplot = plt.imshow(train_images[idx].reshape((28,28)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step to building a simple neural network is to specify layers. The most basic building block is the dense layer (AKA linear layer or fully connected layer), so we'll declare a function that creates the layer. Each dense layer is composed of two variables, the weight matrix `W` and the bias vector `b` as well as a non-linear activation function `a`, to calculate the function `f(x) = a(Wx + b)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dense_layer(x, output_size, activation=tf.identity, name='dense'):\n",
    "    \"\"\"\n",
    "    Args:\n",
    "        - x: a rank two tensor, [batch_size, data_size]\n",
    "        - output_size: (int) number of neurons\n",
    "        - activation: non-linear function applied to the output\n",
    "        - name: TensorFlow name scope for variable\n",
    "    Returns:\n",
    "        a rank two tensor with shape [batch_size, output_size]\n",
    "    \"\"\"\n",
    "    with tf.name_scope(name) as scope:\n",
    "        (_, data_size) = x.get_shape().as_list()\n",
    "        W = tf.Variable(tf.truncated_normal([data_size, output_size]), name='weights')\n",
    "        b = tf.Variable(tf.truncated_normal([output_size]), name='bias')\n",
    "        return activation(tf.matmul(x, W) + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first dimension of the input is reserved to be the \"batch\" dimension, which allows us to run many data through the model simultaneously. The matrix `W` has a row for each input dimension so that each column corresponds to the weights of one linear unit of the layer. After adding the bias vector to the vector resulting from the vector-matrix multiplication, we activate with a non-linearity or the identity function (the latter if we just want to use a linear transformation). [tf.name_scope](https://www.tensorflow.org/api_docs/python/tf/name_scope) is used to group the layer's parameters in TensorFlow's namespace, and its effects can be seen in TensorBoard.\n",
    "\n",
    "TensorFlow variables, which host the model parameters persistently in the graph, are declared with [tf.Variable](https://www.tensorflow.org/api_docs/python/tf/Variable), needing only an initial value to be created. You can also name the variable or set it to be untrainable using optional arguments, but a `Variable` is trainable by default. We use [tf.truncated_normal](https://www.tensorflow.org/api_docs/python/tf/truncated_normal) to provide initial values here to keep it simple, even though there are much better initialization schemes (just make sure you never use a constant, e.g., [tf.zeros](https://www.tensorflow.org/api_docs/python/tf/zeros)).\n",
    "\n",
    "Let's define a simple, two layer network with this function. We activate the first layer with the rectified linear function ([tf.nn.relu](https://www.tensorflow.org/api_docs/python/tf/nn/relu)) and the second layer with [tf.nn.softmax](https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/softmax) so that we can interpret its output as the parameters of a discrete probability distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, [None, 784], name='data')\n",
    "# use a single name scope for the model\n",
    "with tf.name_scope('multi_layer_perceptron_model') as scope:\n",
    "    hidden = dense_layer(x, 200, activation=tf.nn.relu, name='hidden_layer')\n",
    "    output = dense_layer(hidden, 10, activation=tf.nn.softmax, name='output_layer')\n",
    "tf.summary.FileWriter(\"./hackathon2_logs\", tf.get_default_graph()).close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last line, we log what we just defined with TensorBoard for visualization. We use [tf.get_default_graph](https://www.tensorflow.org/api_docs/python/tf/get_default_graph) to retrieve a handle to the TF graph that we're working in (the default because we haven't specified a particular graph), and then we write a summary of the graph with [tf.summary.FileWriter](https://www.tensorflow.org/api_docs/python/tf/summary/FileWriter). This puts an events file in the \"logs\" directory which maybe opened with TensorBoard. Access by running `tensorboard --logdir=./logs` and pointing a browser at http://localhost:6006. Information on how to do this on Crane can be found on Piazza.\n",
    "\n",
    "Note that, though the number of units in the hidden layer may be chosen freely as a hyperparameter, the number of units in the output layer must equal the number of classes in the classification problem. This allows us to use the output as a categorical distribution over the classes, representing the probability that the input belongs to each class.\n",
    "\n",
    "To summarize how the model is performing in its classification task, let's add a placeholder for the correct output and calculate the [cross-entropy](https://stackoverflow.com/a/41990932) loss between the estimated and correct discrete distributions (i.e., between the model's softmaxed distribution and a distribution with a probability of 1 in the correct class, a one hot vector). We use `EPSILON` to avoid potentially trying to calculate `log(0)`, which is undefined."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = tf.placeholder(tf.float32, [None, 10], name='correct_label')\n",
    "EPSILON = 1e-10\n",
    "with tf.name_scope('cross_entropy') as scope:\n",
    "    cross_entropy = -tf.reduce_sum(output * tf.log(y + EPSILON), axis=1)\n",
    "tf.summary.FileWriter(\"./hackathon2_logs\", tf.get_default_graph()).close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've constructed the TF graph, we'll create a session, initialize all variables, and run through an epoch of the test set to plot a histogram of the cross-entropy loss. (One epoch is an iteration of minibatches such that each datum is seen once. Minibatches are also frequently referred to as \"batches\", as below)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAADiZJREFUeJzt3V2MnNV9x/HvrzikDWkDhAVR2+2SxmpDIhWQBbRUURoqXquaSkFyVSVWROVeOCmpIrWQG1dJkIjUBhqpQXJjGidK4yBCi1VQqQVEbS/isLyINxfZAgrGLt7IhkSNktTw78UckwV2PbNmvbve8/1I1j7PmTO7Zx6N97vzzOxsqgpJUn9+bqEXIElaGAZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpU8sWegFHctppp9X4+PhCL0OSjisPPvjg96tqbNi8RR2A8fFxJiYmFnoZknRcSfLfo8zzFJAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdWpR/yawJC0249fdddTXffbGK+dwJW+djwAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMjBSDJnyd5IsnjSb6Z5OeTnJVkR5JdSb6V5MQ29+1tf3e7fHzK57m+jT+V5NJjc5MkSaMYGoAky4E/A1ZX1QeAE4C1wBeAm6pqFXAQuKZd5RrgYFW9F7ipzSPJ2e167wcuA76c5IS5vTmSpFGNegpoGfALSZYB7wD2AR8Gbm+XbwGuattr2j7t8ouTpI1vraqfVNUzwG7g/Ld+EyRJR2NoAKrqBeCvgecYfON/GXgQeKmqDrVpe4DlbXs58Hy77qE2/91Tx6e5zmuSrE8ykWRicnLyaG6TJGkEo5wCOoXBT+9nAb8MnARcPs3UOnyVGS6bafz1A1Wbqmp1Va0eGxsbtjxJ0lEa5RTQ7wHPVNVkVf0fcAfw28DJ7ZQQwApgb9veA6wEaJe/CzgwdXya60iS5tkoAXgOuDDJO9q5/IuBJ4H7gY+0OeuAO9v2trZPu/y+qqo2vra9SugsYBXwvbm5GZKk2Vo2bEJV7UhyO/AQcAh4GNgE3AVsTfL5Nra5XWUz8PUkuxn85L+2fZ4nktzGIB6HgA1V9coc3x5J0oiGBgCgqjYCG98w/DTTvIqnqn4MXD3D57kBuGGWa5QkHQP+JrAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdWqkACQ5OcntSf4ryc4kv5Xk1CTbk+xqH09pc5PkS0l2J3k0yXlTPs+6Nn9XknXH6kZJkoYb9RHA3wL/WlW/AfwmsBO4Dri3qlYB97Z9gMuBVe3feuAWgCSnAhuBC4DzgY2HoyFJmn9DA5Dkl4APApsBquqnVfUSsAbY0qZtAa5q22uAr9XAd4GTk5wJXApsr6oDVXUQ2A5cNqe3RpI0slEeAbwHmAT+IcnDSb6S5CTgjKraB9A+nt7mLween3L9PW1spnFJ0gIYJQDLgPOAW6rqXOB/+dnpnulkmrE6wvjrr5ysTzKRZGJycnKE5UmSjsYoAdgD7KmqHW3/dgZBeLGd2qF93D9l/sop118B7D3C+OtU1aaqWl1Vq8fGxmZzWyRJszA0AFX1P8DzSX69DV0MPAlsAw6/kmcdcGfb3gZ8rL0a6ELg5XaK6B7gkiSntCd/L2ljkqQFsGzEeZ8EvpHkROBp4OMM4nFbkmuA54Cr29y7gSuA3cCP2lyq6kCSzwEPtHmfraoDc3IrJEmzNlIAquoRYPU0F108zdwCNszweW4Fbp3NAiVJx4a/CSxJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnRo5AElOSPJwkn9p+2cl2ZFkV5JvJTmxjb+97e9ul49P+RzXt/Gnklw61zdGkjS62TwCuBbYOWX/C8BNVbUKOAhc08avAQ5W1XuBm9o8kpwNrAXeD1wGfDnJCW9t+ZKkozVSAJKsAK4EvtL2A3wYuL1N2QJc1bbXtH3a5Re3+WuArVX1k6p6BtgNnD8XN0KSNHujPgK4GfgL4NW2/27gpao61Pb3AMvb9nLgeYB2+ctt/mvj01xHkjTPhgYgye8D+6vqwanD00ytIZcd6TpTv976JBNJJiYnJ4ctT5J0lEZ5BHAR8AdJngW2Mjj1czNwcpJlbc4KYG/b3gOsBGiXvws4MHV8muu8pqo2VdXqqlo9NjY26xskSRrN0ABU1fVVtaKqxhk8iXtfVf0xcD/wkTZtHXBn297W9mmX31dV1cbXtlcJnQWsAr43Z7dEkjQry4ZPmdFfAluTfB54GNjcxjcDX0+ym8FP/msBquqJJLcBTwKHgA1V9cpb+PqSpLdgVgGoqu8A32nbTzPNq3iq6sfA1TNc/wbghtkuUpI09/xNYEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4ZAEnqlAGQpE4NDUCSlUnuT7IzyRNJrm3jpybZnmRX+3hKG0+SLyXZneTRJOdN+Vzr2vxdSdYdu5slSRpmlEcAh4BPV9X7gAuBDUnOBq4D7q2qVcC9bR/gcmBV+7ceuAUGwQA2AhcA5wMbD0dDkjT/hgagqvZV1UNt+4fATmA5sAbY0qZtAa5q22uAr9XAd4GTk5wJXApsr6oDVXUQ2A5cNqe3RpI0slk9B5BkHDgX2AGcUVX7YBAJ4PQ2bTnw/JSr7WljM42/8WusTzKRZGJycnI2y5MkzcLIAUjyTuDbwKeq6gdHmjrNWB1h/PUDVZuqanVVrR4bGxt1eZKkWRopAEnexuCb/zeq6o42/GI7tUP7uL+N7wFWTrn6CmDvEcYlSQtglFcBBdgM7KyqL065aBtw+JU864A7p4x/rL0a6ELg5XaK6B7gkiSntCd/L2ljkqQFsGyEORcBHwUeS/JIG/sMcCNwW5JrgOeAq9tldwNXALuBHwEfB6iqA0k+BzzQ5n22qg7Mya2QJM3a0ABU1X8y/fl7gIunmV/Ahhk+163ArbNZoCTp2PA3gSWpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpUwZAkjplACSpU6P8UfgF89gLLzN+3V0zXv7sjVfO42okaWnxEYAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdcoASFKnDIAkdWreA5DksiRPJdmd5Lr5/vqSpIF5DUCSE4C/Ay4Hzgb+KMnZ87kGSdLAfD8COB/YXVVPV9VPga3AmnlegySJ+f+j8MuB56fs7wEumOc1TOtIf3z+SJbaH6Y/2uMAS+9YaPHx/+ncSlXN3xdLrgYurao/afsfBc6vqk9OmbMeWN92PwA8Pm8LPD6dBnx/oRexyHmMjszjM9zxdox+tarGhk2a70cAe4CVU/ZXAHunTqiqTcAmgCQTVbV6/pZ3/PEYDecxOjKPz3BL9RjN93MADwCrkpyV5ERgLbBtntcgSWKeHwFU1aEknwDuAU4Abq2qJ+ZzDZKkgfk+BURV3Q3cPeL0TcdyLUuEx2g4j9GReXyGW5LHaF6fBJYkLR6+FYQkdWrRBsC3jBguybNJHkvySJKJhV7PYpDk1iT7kzw+ZezUJNuT7GofT1nINS6kGY7PXyV5od2PHklyxUKucaElWZnk/iQ7kzyR5No2vuTuR4syAL5lxKz8blWdsxRfonaUvgpc9oax64B7q2oVcG/b79VXefPxAbip3Y/Oac/T9ewQ8Omqeh9wIbChff9ZcvejRRkAfMsIHaWq+nfgwBuG1wBb2vYW4Kp5XdQiMsPx0RRVta+qHmrbPwR2MngXgyV3P1qsAZjuLSOWL9BaFrMC/i3Jg+03qDW9M6pqHwz+cwOnL/B6FqNPJHm0nSI67k9tzJUk48C5wA6W4P1osQYg04z5cqU3u6iqzmNwqmxDkg8u9IJ0XLoF+DXgHGAf8DcLu5zFIck7gW8Dn6qqHyz0eo6FxRqAoW8ZIaiqve3jfuCfGJw605u9mORMgPZx/wKvZ1Gpqher6pWqehX4e7wfkeRtDL75f6Oq7mjDS+5+tFgD4FtGDJHkpCS/eHgbuATfOG8m24B1bXsdcOcCrmXROfxNrflDOr8fJQmwGdhZVV+cctGSux8t2l8Eay9Fu5mfvWXEDQu8pEUlyXsY/NQPg9/o/kePEST5JvAhBu/e+CKwEfhn4DbgV4DngKurqssnQmc4Ph9icPqngGeBPz18rrtHSX4H+A/gMeDVNvwZBs8DLKn70aINgCTp2Fqsp4AkSceYAZCkThkASeqUAZCkThkASeqUAZCkThkASeqUAZCkTv0/k56mCvTCC0MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# calculate values of interest to minibatching and epoch calculation\n",
    "# batch_size is adjustable\n",
    "train_num_examples = train_images.shape[0]\n",
    "test_num_examples = test_images.shape[0]\n",
    "batch_size = 32\n",
    "\n",
    "# finalize the graph\n",
    "with tf.Session() as session:\n",
    "    session.run(tf.global_variables_initializer())\n",
    "    loss_vals = []\n",
    "    # loop through each test datum once, saving the cross entropy\n",
    "    for i in range(test_num_examples // batch_size):\n",
    "        batch_xs = test_images[i*batch_size:(i+1)*batch_size, :]\n",
    "        batch_ys = test_labels[i*batch_size:(i+1)*batch_size, :]\n",
    "        ce_val = session.run(cross_entropy, {x: batch_xs, y: batch_ys})\n",
    "        loss_vals.append(ce_val)\n",
    "\n",
    "# now plot per-datum losses\n",
    "loss_vals = np.concatenate(loss_vals)\n",
    "hist, bin_edges = np.histogram(loss_vals)\n",
    "plt.bar(bin_edges[:-1], hist, width = 1)\n",
    "plt.xlim(min(bin_edges), max(bin_edges))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "It looks like the model guesses the correct class (corresponding to near-zero loss, the smaller bar on the left) on about 10% of the data, which we would have anticipated for a naive model in a 10 class problem. We can improve this model. Let's try this again, this time training with an optimizer.\n",
    "\n",
    "### A Second Attempt...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Clear the graph so we can re-use it. If this is omitted, we get an error.\n",
    "tf.reset_default_graph()\n",
    "x = tf.placeholder(tf.float32, [None, 784], name='data')\n",
    "# use a single name scope for the model\n",
    "with tf.name_scope('linear_model') as scope:\n",
    "    hidden = tf.layers.dense(x, 200, activation=tf.nn.relu, name='hidden_layer')\n",
    "    output = tf.layers.dense(hidden, 10, name='output_layer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, we're using [tf.layers.dense](https://www.tensorflow.org/api_docs/python/tf/layers/dense), which has a very similar API to our custom function, but much more functionality.\n",
    "\n",
    "Further, instead of using a custom cross-entropy function, which has lots of potential for [numerical instability](https://github.com/tensorflow/tensorflow/issues/2462#issuecomment-220842098), we'll use TF's built in function, [tf.nn.softmax_cross_entropy_with_logits_v2](https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits_v2). It combines the `softmax` activation of the final layer with the cross-entropy calculation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# define classification loss\n",
    "y = tf.placeholder(tf.float32, [None, 10], name='label')\n",
    "cross_entropy = tf.nn.softmax_cross_entropy_with_logits_v2(labels=y, logits=output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, using the loss tensor, we'll define an optimizer that uses backpropagation to update the values of each layer's variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# setup optimizer and training operation\n",
    "optimizer = tf.train.AdamOptimizer()\n",
    "train_op = optimizer.minimize(cross_entropy)\n",
    "tf.summary.FileWriter(\"./hackathon2_logs\", tf.get_default_graph()).close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we've chosen the [Adam](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer#minimize) optimizer (usually a safe first choice on any given problem), and use the [minimize](https://www.tensorflow.org/api_docs/python/tf/train/AdamOptimizer#minimize) function which is defined for every TensorFlow optimizer. It returns an operation that automatically calculates the gradient of the provided function, and updates all variables marked trainable. We'll pass it to `sess.run` to train for one epoch and then check the test loss values again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAD5JJREFUeJzt3X+s3XV9x/Hny14Bi+OHoqa2ZK2xcUOTDdIASmIW6/ghxPKHJjWbNoak+4M5dEsc+A+ZPxJIjDiTSdJQXHUMZBVDo0QlgNn8Q6QFJkJ1dIBwAS2mUH+wqa3v/XE+xYve9p5bbu/3sM/zkdzccz7nc07fh7T3ec73nnNIVSFJ6s9Lhh5AkjQMAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktQpAyBJnTIAktSpqaEHOJSTTjqpVq5cOfQYkvSismPHjp9U1avm2jfRAVi5ciXbt28fegxJelFJ8sNx9nkISJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6NdHvBL7v8b2svPSrC3Z7j1xx/oLdliS92PkMQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6NVYAknwoyf1Jvpfk+iTHJFmV5M4kDyb5YpKj2t6j2/ld7fKVM27nsrb+gyTnHJm7JEkax5wBSLIc+BtgTVW9CVgCrAeuBK6qqtXA08BF7SoXAU9X1euBq9o+kpzSrvdG4Fzgs0mWLOzdkSSNa9xDQFPAy5JMAUuBJ4G3AVvb5VuAC9vpde087fK1SdLWb6iqX1bVw8Au4PQXfhckSYdjzgBU1ePAJ4FHGf3g3wvsAJ6pqn1t2zSwvJ1eDjzWrruv7X/lzPVZrvOcJBuTbE+yff+zew/nPkmSxjDOIaATGT16XwW8FjgWOG+WrXXgKge57GDrz1+o2lRVa6pqzZKlx881niTpMI1zCOjtwMNV9VRV/Rq4CXgLcEI7JASwAniinZ4GTgZolx8P7Jm5Pst1JEmLbJwAPAqcmWRpO5a/FngAuAN4V9uzAbi5nd7WztMuv72qqq2vb68SWgWsBr6zMHdDkjRfU3NtqKo7k2wF7gb2AfcAm4CvAjck+Xhb29yushn4QpJdjB75r2+3c3+SGxnFYx9wcVXtX+D7I0kaU0YPzifT0ctW17INn16w23vkivMX7LYkaVIl2VFVa+ba5zuBJalTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTBkCSOmUAJKlTYwUgyQlJtib5fpKdSd6c5BVJbk3yYPt+YtubJJ9JsivJd5OcNuN2NrT9DybZcKTulCRpbuM+A/hH4GtV9UfAnwA7gUuB26pqNXBbOw9wHrC6fW0ErgZI8grgcuAM4HTg8gPRkCQtvjkDkOQ44K3AZoCq+lVVPQOsA7a0bVuAC9vpdcDna+TbwAlJlgHnALdW1Z6qehq4FTh3Qe+NJGls4zwDeB3wFPC5JPckuSbJscBrqupJgPb91W3/cuCxGdefbmsHW3+eJBuTbE+yff+ze+d9hyRJ4xknAFPAacDVVXUq8At+e7hnNpllrQ6x/vyFqk1Vtaaq1ixZevwY40mSDsc4AZgGpqvqznZ+K6Mg/Lgd2qF93z1j/8kzrr8CeOIQ65KkAcwZgKr6EfBYkje0pbXAA8A24MAreTYAN7fT24D3tVcDnQnsbYeIvg6cneTE9svfs9uaJGkAU2Pu+wBwXZKjgIeA9zOKx41JLgIeBd7d9t4CvAPYBTzb9lJVe5J8DLir7ftoVe1ZkHshSZq3sQJQVfcCa2a5aO0sewu4+CC3cy1w7XwGlCQdGb4TWJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVMGQJI6ZQAkqVNjByDJkiT3JPlKO78qyZ1JHkzyxSRHtfWj2/ld7fKVM27jsrb+gyTnLPSdkSSNbz7PAC4Bds44fyVwVVWtBp4GLmrrFwFPV9XrgavaPpKcAqwH3gicC3w2yZIXNr4k6XCNFYAkK4DzgWva+QBvA7a2LVuAC9vpde087fK1bf864Iaq+mVVPQzsAk5fiDshSZq/cZ8BfBr4MPCbdv6VwDNVta+dnwaWt9PLgccA2uV72/7n1me5jiRpkc0ZgCQXALurasfM5Vm21hyXHeo6M/+8jUm2J9m+/9m9c40nSTpMU2PsOQt4Z5J3AMcAxzF6RnBCkqn2KH8F8ETbPw2cDEwnmQKOB/bMWD9g5nWeU1WbgE0ARy9b/XuBkCQtjDmfAVTVZVW1oqpWMvol7u1V9RfAHcC72rYNwM3t9LZ2nnb57VVVbX19e5XQKmA18J0FuyeSpHkZ5xnAwfw9cEOSjwP3AJvb+mbgC0l2MXrkvx6gqu5PciPwALAPuLiq9r+AP1+S9ALMKwBV9U3gm+30Q8zyKp6q+l/g3Qe5/ieAT8x3SEnSwvOdwJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUKQMgSZ0yAJLUqTkDkOTkJHck2Znk/iSXtPVXJLk1yYPt+4ltPUk+k2RXku8mOW3GbW1o+x9MsuHI3S1J0lzGeQawD/i7qvpj4Ezg4iSnAJcCt1XVauC2dh7gPGB1+9oIXA2jYACXA2cApwOXH4iGJGnxzRmAqnqyqu5up38G7ASWA+uALW3bFuDCdnod8Pka+TZwQpJlwDnArVW1p6qeBm4Fzl3QeyNJGtu8fgeQZCVwKnAn8JqqehJGkQBe3bYtBx6bcbXptnawdUnSAMYOQJKXA18CPlhVPz3U1lnW6hDrv/vnbEyyPcn2/c/uHXc8SdI8jRWAJC9l9MP/uqq6qS3/uB3aoX3f3dangZNnXH0F8MQh1p+nqjZV1ZqqWrNk6fHzuS+SpHkY51VAATYDO6vqUzMu2gYceCXPBuDmGevva68GOhPY2w4RfR04O8mJ7Ze/Z7c1SdIApsbYcxbwXuC+JPe2tY8AVwA3JrkIeBR4d7vsFuAdwC7gWeD9AFW1J8nHgLvavo9W1Z4FuReSpHmbMwBV9S1mP34PsHaW/QVcfJDbuha4dj4DSpKODN8JLEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdmhp6gMW08tKvDj0Cj1xx/tAjSBLgMwBJ6taiByDJuUl+kGRXkksX+8+XJI0s6iGgJEuAfwL+HJgG7kqyraoeWMw5hjQJh6HAQ1GSFv93AKcDu6rqIYAkNwDrgG4CMCkmIURGSBrWYgdgOfDYjPPTwBmLPIMmxCREaFIYQw1hsQOQWdbqeRuSjcDGdvaXP7zygu8d8akOz0nAT4YeYhaTOhdM7myDz5UrD3rR4LMdwqTONqlzweLN9ofjbFrsAEwDJ884vwJ4YuaGqtoEbAJIsr2q1izeeOOb1NkmdS6Y3NkmdS5wtsMxqXPB5M222K8CugtYnWRVkqOA9cC2RZ5BksQiPwOoqn1J/hr4OrAEuLaq7l/MGSRJI4v+TuCqugW4Zcztm47kLC/QpM42qXPB5M42qXOBsx2OSZ0LJmy2VNXcuyRJ/+/4URCS1KmJDcCkfmREkmuT7E4yUS9PTXJykjuS7Exyf5JLhp7pgCTHJPlOkv9ss/3D0DPNlGRJknuSfGXoWWZK8kiS+5Lcm2T70PMckOSEJFuTfL/9fXvz0DMBJHlD+2914OunST449FwAST7U/u5/L8n1SY4ZeiaY0ENA7SMj/osZHxkBvGcSPjIiyVuBnwOfr6o3DT3PAUmWAcuq6u4kfwDsAC6ckP9mAY6tqp8neSnwLeCSqvr2wKMBkORvgTXAcVV1wdDzHJDkEWBNVU3Ua9qTbAH+o6quaa/mW1pVzww910ztZ8jjwBlV9cOBZ1nO6O/8KVX1P0luBG6pqn8eci6Y3GcAz31kRFX9CjjwkRGDq6p/B/YMPcfvqqonq+rudvpnwE5G77weXI38vJ19afuaiEceSVYA5wPXDD3Li0GS44C3ApsBqupXk/bDv1kL/PfQP/xnmAJelmQKWMrvvP9pKJMagNk+MmIifpi9GCRZCZwK3DnsJL/VDrPcC+wGbq2qSZnt08CHgd8MPcgsCvhGkh3tHfKT4HXAU8Dn2mGza5IcO/RQs1gPXD/0EABV9TjwSeBR4Elgb1V9Y9ipRiY1AHN+ZIRml+TlwJeAD1bVT4ee54Cq2l9Vf8ro3d+nJxn88FmSC4DdVbVj6FkO4qyqOg04D7i4HX4c2hRwGnB1VZ0K/AKYmN/RAbTDUu8E/m3oWQCSnMjoCMYq4LXAsUn+ctipRiY1AHN+ZIR+Xzu+/iXguqq6aeh5ZtMOF3wTOHfgUQDOAt7ZjrXfALwtyb8MO9JvVdUT7ftu4MuMDo0ObRqYnvEMbiujIEyS84C7q+rHQw/SvB14uKqeqqpfAzcBbxl4JmByA+BHRsxT+0XrZmBnVX1q6HlmSvKqJCe00y9j9A/i+8NOBVV1WVWtqKqVjP6O3V5VE/HILMmx7Zf5tEMsZwODv/Ksqn4EPJbkDW1pLZP3ce7vYUIO/zSPAmcmWdr+na5l9Du6wU3k/xN4kj8yIsn1wJ8BJyWZBi6vqs3DTgWMHs2+F7ivHWsH+Eh75/XQlgFb2iszXgLcWFUT9ZLLCfQa4MujnxdMAf9aVV8bdqTnfAC4rj04ewh4/8DzPCfJUkavHvyroWc5oKruTLIVuBvYB9zDhLwjeCJfBipJOvIm9RCQJOkIMwCS1CkDIEmdMgCS1CkDIEmdMgCS1CkDIEmdMgCS1Kn/A/GZGu9sHk0tAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 32\n",
    "session = tf.Session()\n",
    "session.run(tf.global_variables_initializer())\n",
    "\n",
    "# train for one epoch\n",
    "for i in range(train_num_examples // batch_size):\n",
    "    batch_xs = train_images[i*batch_size:(i+1)*batch_size, :]\n",
    "    batch_ys = train_labels[i*batch_size:(i+1)*batch_size, :]       \n",
    "    session.run(train_op, {x: batch_xs, y: batch_ys})\n",
    "\n",
    "# loop through each test datum once\n",
    "loss_vals = []\n",
    "for i in range(test_num_examples // batch_size):\n",
    "    batch_xs = test_images[i*batch_size:(i+1)*batch_size, :]\n",
    "    batch_ys = test_labels[i*batch_size:(i+1)*batch_size, :]\n",
    "    ce_val = session.run(cross_entropy, {x: batch_xs, y: batch_ys})\n",
    "    loss_vals.append(ce_val)\n",
    "\n",
    "# now plot per-datum losses\n",
    "loss_vals = np.concatenate(loss_vals)\n",
    "hist, bin_edges = np.histogram(loss_vals)\n",
    "plt.bar(bin_edges[:-1], hist, width = 1)\n",
    "plt.xlim(min(bin_edges), max(bin_edges))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just one training epoch has dramatically improved the model's performance. But on how many of the instances is it actually guessing the correct label?\n",
    "\n",
    "## Hackathon 2 Exercise 1\n",
    "\n",
    "Write code to calculate the maximum aposteriori (MAP) estimate of the model on the test data, and compare to the true labels to calculate a confusion matrix with [tf.confusion_matrix](https://www.tensorflow.org/api_docs/python/tf/confusion_matrix). (For the inexperienced, [what is a confusion matrix?](https://en.wikipedia.org/wiki/Confusion_matrix))\n",
    "\n",
    "(Hint #0: Re-use and modify my code from above. Try not to reinvent the wheel, but always remember to cite borrowed code.)\n",
    "\n",
    "(Hint #1: The MAP estimate is just the class whose probability is greatest. I reccomend using [tf.argmax](https://www.tensorflow.org/versions/master/api_docs/python/tf/argmax) with the correct `axis` argument to find this to find the max over the correct dimension of the output.)\n",
    "\n",
    "(Hint #2: tf.confusion_matrix is a function that needs be run in a `session.run` call that returns matrices. Store the resulting matrices in a list and then sum to get the matrix for the full test dataset. Remember to specify the `num_classes` argument.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Coda\n",
    "\n",
    "### Saving and Loading TF models\n",
    "\n",
    "https://www.tensorflow.org/programmers_guide/saved_model, but we'll also discuss this next time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy Broadcasting\n",
    "\n",
    "TensorFlow uses [Numpy broadcasting](https://docs.scipy.org/doc/numpy-1.13.0/user/basics.broadcasting.html) when doing arithmetic with arrays of different shapes.\n",
    "\n",
    "E.g.,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"scalar-matrix addition\")\n",
    "print(np.ones([2,3]) + 1)\n",
    "print(\"a sample vector\")\n",
    "print(np.arange(3))\n",
    "print(\"matrix-vector addition\")\n",
    "print(np.ones([2,3]) + np.arange(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The importance of non-linearities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/img/spiral.1-2.2-2-2-2-2-2.gif\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "# From Colah's Blog, linearly separating spirals with linear transforms and non-linearities\n",
    "HTML('<img src=\"http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/img/spiral.1-2.2-2-2-2-2-2.gif\">')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "TensorFlow 1.12 (py36)",
   "language": "python",
   "name": "tensorflow-1.12-py36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
